The 2015 CHAOS Report has recently been released by the Standish Group. The CHAOS Reports have been published every year since 1994 and are a snapshot of the state of the software development industry. This year the report studied 50,000 projects around the world, ranging from tiny enhancements to massive systems re-engineering implementations. This year the report includes an enhanced definition of success looking at some additional factors which were covered in previous surveys.
The results indicate that there is still work to be done around achieving successful outcomes from software development projects. This table summarises the outcomes of projects over the last five years using the new definition of success factors (on time, on budget with a satisfactory result).
A trend from previous reports that continued in the latest survey is how smaller projects have a much higher likelihood of success than larger ones, as shown in this table.
With the take up of agile development methods over recent years it was possible to compare project outcomes between agile and traditional waterfall projects. Across all project sizes agile approaches resulted in more successful projects and less outright failures, as shown in this table
A key part of the Standish Group analysis over the last 21 years has been the identification and ranking of the factors which work together to make projects more successful. This year’s results show the following list and ranking of factors:
The definitions for these factors are:
Executive Support: when an executive or group of executives agrees to provide both financial and emotional backing. The executive or executives will encourage and assist in the successful completion of the project.
Emotional maturity is the collection of basic behaviors of how people work together. In any group, organization, or company it is both the sum of their skills and the weakest link that determine the level of emotional maturity.
User Involvement: takes place when users are involved in the project decision-making and information-gathering process. This also includes user feedback, requirements review, basic research, prototyping, and other consensus-building tools.
Optimization is a structured means of improving business effectiveness and optimizing a collection of many small projects or major requirements. Optimization starts with managing scope based on relative business value.
Skilled staff are people who understand both the business and the technology. A skilled staff is highly proficient in the execution of the project’s requirements and deliver of the project or product.
SAME is Standard Architectural Management Environment. The Standish Group defines SAME as a consistent group of integrated practices, services, and products for developing, implementing, and operating software applications.
Agile proficiency means that the agile team and the product owner are skilled in the agile process. Agile proficiency is the difference between good agile outcomes and bad agile outcomes.
Modest execution is having a process with few moving parts, and those parts are automated and streamlined. Modest execution also means using project management tools sparingly and only a very few features.
Project management expertise is the application of knowledge, skills, and techniques to project activities in order to meet or exceed stakeholder expectations and produce value for the organization.
Clear Business Objectives is the understanding of all stakeholders and participants in the business purpose for executing the project. Clear Business Objectives could also mean the project is aligning to the organization’s goals and strategy.
InfoQ spoke to Jennifer Lynch from the Standish Group about the results:
InfoQ: There is a very significant change in the definition of success in this year’s report with six factors now being included in the overall measure of success (on Time, On Budget, on Target, on Goal, Value and Satisfaction) Why did you add the additional factors and how does this influence the way readers should think about project outcomes?
Jennifer: The Standish Group has redefined project success as onTime, onBudget with a satisfactory result. Success is hard to define and we had a hard time coming to this conclusion. Merriam-Webster dictionary defines success as the fact of getting or achieving wealth: respect or fame: the correct or desired result of an attempt; someone or something that is successful: or a person or thing that succeeds. The Project Management Institute (PMI) has defined success as onTime, onBudget, and onTarget also known as the Triple Constraints and the Iron Triangle. However, we have seen many projects that have met the Triple Constraints and did not return value to the organization or the users and executive sponsor were unsatisfied.
More details can be found in this blog post which explains rationale behind the change.
InfoQ: Reading the report, it seems that to have a strongest likelihood of a successful project you need to buy a off the shelf product or modernise an existing system. How would you explain this? Does that mean that all the noise around agile / lean start up practices have not achieved their goal?
Jennifer: The two data points are mutually exclusive, you can use an agile method to modernize or implement a package. In fact, the database has those types of projects.
InfoQ: The results include a complexity index – how does complexity impact the success of projects?
Jennifer: The table below shows the size/complexity matrix. The more complex and bigger the higher the risk of failure.
InfoQ: Does using an agile approach help overcome complexity and why would that be?
Jennifer: It can by failing earlier and restarting faster.
InfoQ: One of the success factors listed is higher “emotional maturity skills” – please can you explain what this is and how it seems to be impacted by geographic region?
Jennifer: Emotional maturity skills are often called the soft skills. Emotional maturity skills include the 5 deadly sins of PM , managing expectations, consensus building, and collaboration. We have identified 50 emotional maturity skills and have tested these skills throughout the world. We saw a major correlation between poor emotional maturity skills and lower success/value rates. We have several educational services around Emotional maturity skills including: low cost reports, appraisals/benchmarks and workshops.
InfoQ: Another success factor is the skill level of the team members – are organizations investing enough in increasing the skills of their people?
Jennifer: No, this is real shortfall. Moving investments from PM tools and other worthless activities should improve success and value rates. Everyone is looking for a quick fix, but investing in people takes time, but offers a much bigger payout in the end.
InfoQ: The results indicate that a small proportion of organizations are assigning return on investment values across project requirements. How can organizations assign value without looking at ROI across the separate requirements?
Jennifer: Developing ROIs is hard and costly. We suggest our Value Portfolio Optimization and Management Service. ROIs are assigned by relative value, using a scale from 1 to 5 requirements can be categorized and then optimized.
InfoQ: The figures about goal setting seem counter intuitive, in that projects with precise goals are less successful than ones with vauge goals. Does this validate the hypothesis of Takeushi & Nonaka about the knowledge acquisition process (the sponsor / Director has to set a sufficiently vague goal definition to the team to ensure that the team can translate that into their own vision).
Jennifer: We would support Takeushi & Nonaka hypothesis and go a little further. Only that which is unknown produces real value. Having precise goals means you know everything in advance, which is usually not the case. IT is also always trying to align itself with the organization. However as Chairman Jim Johnson would say, if IT and the organization are aligned, one or both are not going fast enough.
InfoQ: Does the definition of agile used in the report include non-timeboxed approaches such as Kanban, lean practices and continuous delivery?
Jennifer: We would include them. Agile to us is broad term without regard to a particular methodology such as Scrum.
InfoQ: The report is focused on projects, yet many organizations are moving away from project-based work to product-line activities with stable teams – how is that factored into your results?
Jennifer: We have been a long proponent of pipeline driven or continuous flow software development. We do think this is the correct approach. However, our numbers do not include this approach since we are collecting project profiles. However we are looking at ways to capture this information for future reporting.
InfoQ: Will there be a CHAOS Report 2016, and beyond that?
Jennifer: A. The next CHAOS Report will be published in April 2016.
InfoQ: Where can InfoQ readers find more details about the Standish Group and the Chaos reports?
Jennifer: We run 4 websites for information on the organization:
About the Authors
Shane Hastie is the Chief Knowledge Engineer for Software Education a training and consulting company working in Australia, New Zealand and around the world. Since first using XP in 2000 Shane's been passionate about helping organisations and teams adopt Agile practices. Shane leads Software Education's Agile Practice, offering training, consulting, mentoring and support for organisations and teams working to improve their project outcomes.In 2011 Shane was elected as a Director of the Agile Alliance.
Stéphane Wojewoda is actually working as Lead IT PMO and coach for teams in agile / lean transition in a large retail French Company. Stéphane is dedicated to promoting humanism in organisations through the Agile & Lean values and mindset, letting project or product teams choose what they think is best for them.
